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Abstract 

The purpose of this study was to identify the strategies used by learners in an open-ended hypermedia 
information system. Four participants were drawn from an introductory technology course incorporating a unit on 
telecommunications. Participants completed a survey measuring reported knowledge in three domains (metacognitive, 
system, and subject) as well as self-efficacy toward technology. Participants searched for information using Netscape®, 
thinking aloud as they searched. Data were collected; analysis oecured in several phases: scripting the search, reading 
through the data, segmenting according to research question, and encoding. Three major findings related to hypermedia 
information systems resulted from this study: a variety of strategies are used by learners; reported knowledge does have 
an affect on the strategies used; and perceptions of disorientation and level of perceived self-efficacy have an affect on the 
strategies used. Implications related to emerging information technologies, open-ended learning environments, and the 
ways learners think are considered. 



INTRODUCTION 

Interest in creating productive and stimulating learning environments is an enduring theme in the educational 
arena. Recent attention has focused on open-ended learning environments (OELEs). In OELEs, experience and context are 
critical for cultivating cognitive processes that support understanding. These environments are learner-centered, with the 
instructor adopting the role of facilitator as opposed to presenter. The learner is encouraged to explore and experiment; 
problem solving, critical thinking, and perspective-building are inherent processes in these environments. While learning 
outcomes remain important to knowing, processes are the focal point of growth in understanding (Hannafin, Hall, Land, 
& Hill, 1994; Papert, 1993). 

The emphasis on cultivating processes and strategies increases the requirement for learner-supplied structure. 
While the instructor or system can assist by guiding and facilitating, the learner determines which tools and resources to 
access and manipulate; that is, how to navigate to promote learning (Bransford et ai., 1986; Spiro, Feltovich, Jacobson. 
& Coulson, 1991). The individual determines what s/he wants or needs to know, for accessing the information in the 
system, and for deciding whether or not the system contains the needed information (Perkins, 1993; Roth & 
Roychoudhury, 1993). 

Numerous information systems exhibit some characteristics cf OELEs. These systems blur traditional 
distinctions between educational systems and information retrieval systems. They have become increasingly transparent 
and user-centered. Emerging information systems provide the user with more control through “point and click” 
technology, and expand the retrievable resources to include multiple media. 

Emerging information systems do more than archive and retrieve data; they support the individual's efforts to 
learn, think, and understand (Hannafin, 1992). These systems are no longer the exclusive domain of information 
scientists. They are readily accessible and usable by the consumer, creating both extraordinary opportunities for student- 
centered learning, as well as formidable challenges for those who design and develop such systems. 

The challenges associated with creating electronic information systems that function as OELEs are considerable. 
Some systems are simplistic and focused; others are complicated and virtually limitless in breadth and depth. Designing 
and developing electronic information systems that empower the user, are intuitive and self-evident, and inclusive in 
orientation is a formidable challenge (Norman, 1988). It has also proven difficult to assist and support the learner in 
their uniquely individual goals as they use such systems. Learners accustomed to interacting in environments that are 
prescriptive and directed are often inefficient arid ineffective in their use of and interactions within open-ended, learner- 
centered environments. They may lack the orientation, mental model, or strategies necessary for the environments (Hill. 
Lebow, Driscoll, & Rowley, 1994). 

Another concern is that emerging information systems have evolved despite the absence of a strong theoretical 
or research foundation. We lack a solid understanding as to how or why the systems are being used. In addition, we lack 
a clear conceptualization of the capabilities and/or features that are needed or desired. 

The purpose of this study was to identify the strategies used by learners seeking information in an open-ended 
hypermedia information system. The following questions provided a framework for this study. 

Primary Question 

What strategies are employed in open-ended hypermedia information systems? 

Secondary Questions 

1 . Does metacognitive knowledge affect the strategies employed? 

2. Are perceptions of disorientation influenced by the strategies employed? 

3. Does perceived self-efficacy affect the strategies employed? 

4. Does system knowledge affect the strategies employed? 
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5. Does subject knowledge affect the strategies employed? 



METHODOLOGY 

Selection and Description of Participants 

The population of this study were current and prospective educators enrolled in a university-level technology for 
educators course. This included Pre-K through 12 public school teachers as well as college/university instructors. This 
sample and population was chosen for several reasons: a lack of research examining adult use of hypermedia information 
systems, willingness to participate in an intensive study, and diversity of the participants. The university serves both 
undergraduate and graduate students from the state, region, and nation, as well as international students. The students 
reflected a broad range of experiences and backgrounds, assuring diversity in the population. 



The Course: Technology for Teachers 

•Technology for Teachers” attempted to create the conditions under which people typically understand things: 
first they develop a need to leam, then subsequently satisfy the need through progressive experiences. The course 
operated under several important premises. First, within educational settings, technologies are not so much a curriculum 
as sets of tools which can be put to productive uses. The course worked to help learners to establish what those 
productive uses might be. 

Another premise was that productive use is influenced by context, audience, and activity. Problems presented 
during the course were anchored in everyday classroom teaching and learning contexts for which various technologies can 
be employed for, and with, students, teachers, and communities. The goal was not to become an expert in technology, 
but to become more facile in teaching through technology. 

The final premise was that it was critical to understand the processes associated with various technology 
applications. Technology continues to advance at an exponential rate. Existing technologies will inevitably be replaced 
by systems that are faster, and more powerful. Therefore, problem solving and cognitive strategies for the use of 
technology were highlighted, rather than mastery of specific software and hardware. 

Technology for Teachers was chosen for several reasons: 

• Since the course was an elective, most students, both undergraduates and graduates, entered with a high 
level of interest and motivation. 

• Given the recent movement to integrate technology into the classroom, the learners entered with a high 
degree of interest in learning various technologies, as well as diverse techniques for applying technology in 
their individual settings. 

• The course itself reflected the assumptions and principles associated with open-ended learning 
environments (OELEs), including the development of cognitive strategies and higher-order thinking skills 
(sec, for example, Hannafin, Hall, Land, & Hill, 1994). An inherent aspect of OELEs is that the learners 
establish goals they want to accomplish, as well as determine the steps needed to attain the goals. The 
course naturally supported the participants in the use of hypermedia information systems. 

• Telecommunication technologies were integral to the course activities. A major focus of the unit was the 
Internet. Most learners were aware of the Internet, even if they did not understand what it was or how it 
could be used in an educational setting. The majority had a high degree of curiosity and motivation to learn 
about the Internet , what it can do, and the ways in which it can be used. 

The Information System: Netscape® 

The system selected for the study was Netscape®, a hypermedia information system built upon World Wide 
Web(W3) technologies. The system displays the following characteristics: 

• open-ended in nature, providing access to a variety of information and media 

• requires generative activities, placing the learner in the driver’s scat 

• necessitates an orientation toward discovery, where the learner is free to explore vast resouces 

• control of the environment centers around the user, both in terms of the information requested and the 
steps taken to retrieve the information 

The World Wide Web is an interconnection of computer sites on the Internet. The main difference between 
World Wide Web technologies and traditional Internet programs is that W3 allows for the transmission of pictures, sound 
and motion along with traditional textual information. It also differs from traditional Internet programs in that it creates 
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a point and click interaction environment, where a mouse or track ball can be used to retrieve information. This creates 
an environment where hypertext and/or hypermedia capabilities are readily accessible. 

Netscape® is a front-end browser used to interface with the Web. Netscape® and W3 operate in a client/server 
environment: the program that resides on the local computer is the client, while the server is the remote location 
accessed. Netscape® is the browser that displays the information retrieved from the Web. 

Method and Design 

An embedded case study approach was employed, involving the use of multiple cases, or embedded units, within 
a larger context. The unit of study in this case was the individual user of the system from whom several data sources 
were gathered. A unique strength of the case study strategy is the ability to deal with multiple sources of evidence. 
Various sources of evidence from the embedded units allowed for triangulation of the data, as well as serving to address 
concerns with construct validity (Yin, 1994). 

The methods combined exploratory, descriptive, and analytical approaches. The study was exploratory in that 
several of the methods have not been previously used to examine the use of strategies in hypermedia information 
systems. The research was descriptive in that the goal was to describe the strategies employed from the evidence 
gathered, including the perspectives of the participants and the researcher. Information gathered during the research was 
manipulated iteratively through analytical induction. Initial definitions and explanations were developed at the beginning 
of the study. These definitions and explanations were then contrasted with the data collected and refined based on analysis 
of the evidence (Bogdan & Biklen, 1992). 



Measures and Instrumentation 

A combination of rationalistic and naturalistic techniques was used. Rationalistic (quantitative) techniques, 
including surveys and questionnaires, were used to generate individual difference measures for each case, and audit trails 
were used to track the participants 1 movements as they searched using Netscape®. Naturalistic (qualitative) techniques, 
included think-aloud protocols and interviews. These methods were used to monitor the participants 1 search, as well as to 
capture their thoughts both during and after the search process. 

A fundamental theme in naturalistic studies is the element of continuous improvement. It is a widely held 
belief in qualitative research that research is a process that occurs on a continuum, one moving ever closer to a contextual 
reality. In keeping with this theme, and to assist the researcher in creating a more complete picture of contextual reality, 
the techniques described below were first used in a developmental study. 

Figure 1 illustrates the progression of the activities throughout the study. Several activities took place during 
the study, which can be divided into three main components: pre-search activities, activities during the search, and post- 
search activities. Five techniques were used to gather data: pre-search surveys, think-aloud protocols, audit trails, post- 
search questionnaires, and stimulated post-search interviews. 

Pre-search survey . The pre-search survey measured the learners intial perceptions of the three domain areas 
(metacognitive, subject, system) as well as their perceived self-efficacy. 

Think-aloud protocols . Think-aloud protocols were recorded on both audio and video tape as the learners worked 
in the hypermedia information system. Learners were encouraged to verbalize whatever came to mind as they worked in 
the system. 

Audit trails . Video was used to record the learner’s path in the World Wide Web. The recordings were used to 
generate audit trails of the search. 

Post-search questionnaires . Like the pre-search survey, the post-search questionnaire measured the learners 
perceptions of the three domain areas (metacognitive, subject, system) as well as their perceived self-efficacy. In addition, 
the questionnaire examined the learner’s perceived strategies as they worked in the system. 

Stimulated post-search interviews . Following the transcription of protocols and the creation of audit trails, 
learners wae interviewed to gamer their reflections, reactions, and comments on the search task. 

See Appendix B of the 1996 AECT Proceedings 

The techniques enabled the researcher to establish a rich orientation to the individual’s search process, procedures 
used, and strategies engaged. Those techniques were used to determine the five self-reported knowledge areas: 
disorientation, perceived self-efficacy, metacognitive knowledge, system knowledge, and subject knowledge. 




273 



o 



Disorientation - Disorientation refers to a loss of one’s bearings. Disorientation is often associated with a lack 
of understanding, or an inability to recognize how to use the system to accomplish one’s goals. 

Perceived self-efficacv . Self-efficacy refers to a person’s perceptions of ability to succeed on a given task. 
Perceived self-efficacy refers to a person’s judgments of her or his capabilities to organize and execute action required to 
attain designated performance (Bandura in Olivier & Shapiro, 1993). Perceived self-efficacy transcends knowing what to 
do or awareness of one’s skills; it is a judgment of how well one can perform using particular knowledge or skill. 

Mftarnynitive knowledge Metacognition refers to awareness about cognition. Metacognitive knowledge 
enables an individual to reflect, evaluate and direct cognitive activities more effectively (Perkins, Simmons, & Tishman, 
1990). In this study, metacognitive knowledge referred to the participant’s self-reported awareness of their cognitive 
processes. Users with extensive metacognitive knowledge were presumed able to monitor their learning needs, leading to 
purposeful system use. 

System knowledge . System knowledge refers to the participant’s reported knowledge about electronic 
information systems in genera], as well as Netscape®, in particular. System knowledge was felt to influence the ability 
to search successfully in an information system. Users with extensive system knowledge, such as reference librarians, 
are often able to use it to maneuver through information systems, despite a lack of significant prior subject or domain 
knowledge. 

Subject knowledge . Subject knowledge refers to estimates of existing knowledge and experience related to the 
subject for which searches were conducted. Subject knowledge was presumed to influence the strategies participants 
employed as they searched using Netscape®. 

METHODS, ANALYSIS AND PROCEDURES 

Data analysis was conducted at several stages. The developmental study provided an opportunity to test the 
clarity of the questions and terminology, the techniques employed with the think-aloud protocol, and the accuracy of the 
technology employed to gather data. It also provided insight into the strategies being employed as people work in these 
systems. 

Primary Research Phase 

Preliminary Organization and Analysis 

Most of the collection, organization, and analysis of data occurred concurrently. This helped to indicate gaps in 
Hata as they were gathered and organized, allowing adaptations in the process and indicating the need for .additional 
information (see, for example, Glaser & Strauss, 1967; Hert, 1992). Formal analysis, comprising several reviews of the 
data, took place toward the end of data collection (Bogdan & Biklen, 1992). Overall organization of the data gathered in 
relationsh'p to the research questions is illustrated in Table 1 . 



Table 1. Research question, data sources, and methods 



Research Question 


Data Gathered 


How Obtained 


When 


Who 


1. Docs metacognitive 


Written responses 


Pre-Search Survey 


Pre-search 


Participant 


knowledge affect the 


to several questions 








strategies employed? 


Protocol 


Think-Aloud 

Protocol 


Search 


Participant 




Audit Trail 


Video tape 


Search 


Participant 




Written responses 
to several questions 


Post-search 


Post- 


Participant 






Question. 


search 






Verbal responses to 


Stim. Post-Search 


Post- 


Participant 




several questions 


Inter. 


search 


& 

Researcher 



t. 
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Table 1. (continued) 



Research Question 


Data Gathered | 


How Obtained | When 1 


Who 


2. Are perceptions of 


Protocol 


Think-Aloud 


Search 


Participant 


disorientation influenced 




Protocol 






by the strategies 


Audit Trail 




Search 


Participant 


employed? 




Video tape 








Written responses 




Post- 


Participant 




to several questions 


Post-Search 


search 








Question. 




Participant 




Verbal responses to 




Post- 


and 




several questions 


Stim. Post-Search 


search 


Researcher 






Inter. 






3. Does perceived self- 


Written responses 


Pre-Search Survey 


Pre-search 


Participant 


efficacy affect the 


to several questions 








strategies employed? 




Post-search Ques. 


Post- 


Participant 




Written responses 




search 






to several questions 


Stim. Post-Search 




Participant 






Inter. 


Post- 


and 




Verbal responses to 




search 


Researcher 




several questions 








4. Does system 


Written responses 


Pre-Search Survey 


Pre-search 


Participant 


knowledge affect the 


to several questions 








strategies employed? 




Think-Aloud 


Search 


Participant 




Protocol 


Protocol 












Search 


Participant 




Audit Trail 


Video tape 






5. Does system 


Written responses 


Post-search 


Post- 


Participant 


knowledge affect the 


to several questions 


Questionnaire 


search 




strategies employed? 












Verbal responses to 


Stimulated Post- 




Participant 




several questions 


Search Interview 


Post- 


and 








search 


Researcher 


6. Does subject 


Written responses 


Survey 


Pre-search 


Participant 


knowledge affect the 


to several questions 








strategies employed? 












Protocol 


Think-Aloud 


Search 


Participant 






Protocol 








Audit Trail 




Search 


Participant 






Videotape 








Written responses 




Post- 


Participant 




to several questions 


Post-search 


search 








Questionnaire 








Verbal responses to 






Participant 




several questions 


Stimulated Post- 


Post- 


and 






Search Interview 


search 


Researcher 




► / 

( 
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Table 1. (continued) 



Research Questions 


Data Gathered 


How Obtained 


When 


Who 


Overall Question: 


Written responses 


Survey 


Pre-search 


Participant 


What strategies are 


to several questions 








employed in open-ended 










hypermedia information 


Protocol 


Think-Aloud 


Search 


Participant 


systems? 


Audit Trail 


Protocol 


Search 


Participant 






Videotape 








Written responses 




Post- 


Participant 




to several questions 


Post-Search 


search 








Questionnaire 








Verbal responses to 






Participant 




several questions 




Post- 


and 






Stimulated Post- 


search 


Researcher 






Search Interview 







Transcripts of the think-aloud protocols, as well as the video-taped search sessions, were generated. Think-aloud 
protocol transcripts provided a verbative translation of the audio tape. Transcription of the video tapes involved a step- 
by-step recounting of the participant’s movement through the information space. The information gathered from the 
video tapes was transformed into a search trail generated using the program Inspiration®. Locations visited by the 
participant during the search are represented as nodes in the map. 

Preliminary analysis took place as transcriptions were generated. A participant summary sheet was used to 
summarize this analysis and to assist in organizing the data (McGregor, 1993; Miles & Huberman, 1984). The summary 
sheet addressed each research question according to the instrument examined, directing the researcher to summarize the 
main points from the survey in relation to: metacognitive knowledge, perceived self-efficacy, system knowledge, and 

subject knowledge. . 

Stimulated post-search interviews were scheduled one week after completion of data collection. Initial analysis 
of the think-aloud protocols and search trails helped to guide the formulation of additional questions appropriate for 
individual interviews. Video tapes were shown during the interview to stimulate the participant’s recall of particular 
instances in the search process. 

Data were examined with an orientation toward both breadth and depth. In order to obtain breadth and depth, the 
initial pool of 14 participants was reduced to four for in-depth analysis. The researcher reviewed the participants’ data in 
relation to the research questions, the performance of each participant in the think-aloud process, and the need to represent 
the range of participants in the study. The four participants chosen for in-depth analysis were Mick, Bill, Marsha, and 
Alyssa. As shown in Table 2, each reflected different estimates of the various knowledge described previously. The high, 
mid, and low designations for the participants reflects an overall ranking (“average”) based on level of disorientation, 
perceived self-efficacy, and self-reported knowledge in the areas of metacognition, system and subject. Names of the 
participants were changed to protect their identify. 



Table 2. Summary of participants reported knowledge, 
overall ranking, and “success” 



Parti- 

cipant 


Metacog- 

nitive 


Disorien- 

tation 


Self- 

Efficacy 


Sys- 

tem 


Sub 

-ject 


Overall 

Ranking 


Success 

Rating 


Alyssa 


mid 


high 


low 


low 


mid 


low 


no 


Bill 


high 


mid 


mid 


low 


mid 


mid 


yes 


Marsha 


low 


mid 


mid 


low 


low 


mid 


no 


Mick 


Ml 


low 


—M 


low 


Jlifiii 


high 


yes 
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In-Depth Analysis 

Given the variety and volume of evidence gathered, techniques of examination, categorization, tabulation and re- 
combination were used throughout the analysis. Copies of original data sets, as well as records of each transformation, 
were used to maintain the integrity of the data during analysis iterations. The data were reviewed and triangulated several 
times by the researcher; each time the data were examined, additional relevant information and related findings were 
revealed. 

The initial in-depth analysis was driven by the need to combine the think-aloud protocols and the search trails. 

A “script” format, often used in screen plays, movies, and television was used to merge the elements. The scripts were 
used throughout the remainder of the analysis to aid the researcher in identifying themes, patterns, and strategies. 

The next round of analysis involved several stages, including reading through the data, highlighting instances in 
the data related to the research questions, and generating case report for individual participants (Bogdan & Biklen, 1992; 
Harmon, 1992; Yin, 1994). “Reading through the data” involved a review of all data elements, “reading” the instruments 
like a book. Initial themes and patterns were identified. 

Data were marked-up (segmented) in accordance with the research questions during the next phase of analysis. As 
the researcher read the data, multi-coloured highlighters were used to mark-up the data according to research questions (see, 
Ericsson & Simon, 1984, for a description of the mark-up process). 

The sections of the scripts related to specific research questions were then coded. A mixture of established and 
data-driven codes were used during this stage. The “established” codes were derived from other information seeking 
studies, including information-processing research, information-seeking models, and interactive strategy research. These 
codes relate to strategies and actions taken by users of information systems. Themes and patterns not readily applicable 
to established categories were assigned a new category. 

At this point, the researcher asked participants to confirm the validity of themes, patterns, and strategies (Bogdan 
& Biklen, 1992; Mathison, 1988). Inclusion of the participant in the research process is a fundamental element in 
qualitative research. To facilitate feedback from each participant, individual case reports were generated. Each participant 
was asked to review the report and to generate questions or concerns for discussion with the researcher. Verification from 
the participant is valuable in improving the validity and reliability of a study. 

One week after the reports were distributed, the researcher met with each participant to discuss the case reports. 
Results of these routings indicated a high degree of researcher-participant agreement. Recommendations from each 
participant were noted during the interviews; where indicated, changes and/or additions were made to the data sets for each 
participant. 

The final stage of analysis involved the collation of data according to the corresponding research questions. Each 
data sou rce, organized according to research questions, was re-analyzed to match and clarify patterns. Pattern matching 
involve i examination for similarities in the thoughts and/or actions of the participants. Codes were assigned to the 
strategics in the transcribed protocol. Examination of the coded protocol was undertaken to identify trends in the data. 

This p ocess, repeated for each participant as well as across participants, enabled the researcher to identify trends in 
strategies as they related to each research question. 

In order to interpret the meaning of participants’ actions, the technique of explanation building was used. 
Explanation building involves returning to the data to seek explanation for why participants thought or acted during the 
search process. Returning to the data for explanation helped to establish the creation of assertions related to the study. 

Analysis of global patterns and trends was necessary to address the overall research question. To assist the 
researcher in theorizing on a global level, two methods of aggregation were used (Ericsson & Simon, 1984); aggregation 
by episode and aggregation by process. In aggregating the data, the researcher merges individual cases into one instance. 
In this study, aggregation was completed around two data points: similar episodes in each search and similar processes 
used by each learner. 

The final stage of the analysis process involved cumulative data analysis, where all trends, issues, and themes 
generated during the analysis are examined as complete units relating to each research question. From these data, the 
researcher generated speculations and explanations of the strategies used by learners as they search in hypermedia 
information systems. 



FINDINGS AND RESULTS 

The results of the study were constructed from the theoretical foundations of grounded tieory. Grounded theory 
involves building explanations from the data themselves (Glaser & Strauss, 1967). Pattern matening techniques were 
employed during the iterative analysis process as a way of making sense of the data. Analysis of the data yielded several 
assertions. 
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Assertions By Research Area 

Assertion 1 : Perceptions of disorientation affect the strategies employed in a hypermedia information system. 
Several learners reported feeling “lost,” or “being in the middle of nowhere" when they were using the system. This 
disorientation, in turn, affected search decisions. 

Assertion 2: Prior subject knowledge is directly related to concept recognition and affects the strategies 
employed. Learners with high subject knowledge readily recognized terms related to the search topic. This impacted what 

they pursued in the system and how they used the program. 

Assertion 3: System knowledge has direct impact on the strategies the learner is able to develop as they seek 
information in a hypermedia information system. What the user is able to do with the system is directly tied to their 
knowledge and experience with the system. Development of strategies takes time; learners with higher system 

knowledge were able to develop strategies easier and quicker. 

Assertion 4: Level of metacognitive knowledge influences strategy refinement. Learners with high levels of 
metacogr.itive knowledge were able to reflect on their process. This reflection enabled them to refine their actions and 
make better use of the system. 

Assertion 5: Self-efficacy affects willingness to engage in hypermedia exploration. Learners with high levels of 
self-efficacy engaged in more exploration of the system. This increased exploration afforded more opportunities, 
increasing prospects of finding the desired information. 

Results By Research Questions 

Overall Research Question : What strategies are employed in oi<en-ended hypermedia information systems? 

This “meta-question” was informed by the secondary questions. However, the findings can be summarized into 

three trends: 

• A variety of strategies are used by learners as they seek information in a hypermedia information 
system. 

• Metacognition, system and subject knowledge affect the strategies used. 

• Perceptions of disorientation, as well as perceived self-efficacy, affect the strategies used. 

Specific findings are presented as they relate to each research question. 

Research Question One: Does metacognitive knowledge affect the strategies employed? 

Of the prior knowledge areas examined, metacognitive knowledge most influenced the strategies used while 
searching using Netscape®. Level of metacognitive knowledge seemed to corresponded to success in system use. 

Learners who used the system effectively were active information processors and comprehended who monitored their 
learning activities (Osman & Hannafin, 1992). They also experienced less disorientation. Mick consistently monitored 
his thinking and revised his actions accordingly, and also made significant progress with the strategies used and 
understanding of the search process. Mick engaged in several metacognitive tasks described by Wang, Haertel, and 
Walberg (1990): comprehension, monitoring, use of self-regulation, self-control strategies, and use of the strategies to 
facilitate generalization. 

All learners reported metacognitive experiences during their search tasks. According to Flavell (1979), a 
metacognitive experience is any conscious cognitive experience that accompanies and pertains to an intellectual 
enterprise. His illustration of a metacognitive experience directly relates to what the participants experienced while 
searching using Netscape®: “To illustrate, you may experience a momentary sense of puzzlement that you subsequently 
ignore, or you may wonder for some time if you really understand what another person is up to” (p. 908). Both the sense 
of puzzlement and wondering about understanding were experienced by all of the learners in the study. 

Interestingly, the manner in which metacognitive experiences were processed influenced whether, or how, 
metacognition infli enced the search task: 

Somr metacognitive experiences are best described as items of metacognitive knowledge that have entered 
const iousness. As one example, while wrestling with some stubborn problem, you suddenly recall another 
probl ;m very like it that you solved thus and so. Some metacognitive experiences clearly cannot be 
described that way, however. For instance, the feeling that you are still far from your goal is not in itself a 
segment of metacognitive knowledge, although what you make of that feeling and what you do about it 
would undoubtedly be informed and guided by your metacognitive knowledge... (Flavell, p. 908). 
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This differentiated Mick from the other learners: not only did he have prior experience, his actions were informed and 
guided by his high level of metacognitive knowledge. 

While the importance of metacognitive knowledge is clear, the implications are not. Some advocate teaching 
metacognition knowledge for skill (Osman & Hannafin, 1992; Wang, Haertel, & Walberg, 1990); others have trained 
students to effectively monitor their learning (Brown & Palincsar, 1989; Paris, Cross, & Lipson, 1984). Still, the 
results are inconsistent; recommendations for teaching or training are often suspect. While results of this study confirm 
its importance, further investigation is needed to determine the best ways to instruct learners in the use of these skills. 

Research Question Two: Are perceptions of disorientation influenced by the strategies employed? 

The perception of disorientation yielded one of the strongest influences on the strategies employed. liigh levels 
of disorientation can lead to dysfunction as well as overall dissatisfaction with the search process. This is not only 
debilitating, but also serves to elevate frustration. 

Alyssa's high level of disorientation and dissatisfaction with the search process provides the most compelling 
evidence for this conclusion. Her failure to retrieve relevant information, in addition to her use of low-end, basic 
strategies, suggests disorientation has a seriously detrimental influence in the use of emerging information systems. 

This study provided more evidence for the phenomenon: “lost in hyperspace.” This occurred independent of the 
level of reported knowledge. Several researchers have discussed its negative consequences on usability (see, for example, 
Jonassen & Grabinger, 1989; Marchionini, 1988). Newby (1992) proposed a way to help orient the user, reducing the 
perceptions of disorientation: involve the user directly in the retrieval process. Newby's “navigable systems” Gpen up 
the black box, taking the user “behind the scenes” instead of leaving them “in front of the curtain,” waiting for the 
system to return results. According to Newby, easily navigable systems enable the user to directly engage in the retrieval 
process, which, in turn, reduces the level of disorientation. 

Marchionini's (1993) techniques of zooming and browsing are similar to Newby's navigable system space. 

Like navigable systems, zooming and browsing involve the user directly in the retrieval process: the system zooms into 
a relevant area of the information space, offering several suggestions that may or may not be relevant to the learner's 
needs. After moving closer, the learner is better able to browse and select the desired information from what is available. 
The system does not supplant entirely the effort of the learner, but instead engages them directly in the process of 
determining relevance. 

In some ways, Netscape® provides zooming and browsing capabilities for the World Wide Web. The learner 
submits a request, after which the search engine scans the Web information space an^ .etums a list of “hits.” At this 
point, the concepts of zooming and browsing break down in several important ways. A primary way that it breaks down 
is in what is returned. As currently structured, the listing of relevant documents can range from titles of the documents 
to http:// addresses to ftp sites. Obviously, this makes the list less intuitive and transparent. 

One way to overcome this problem might be to standardize how the systems provide feedback to the learners. 
Several systems have already been developed, such as the standardized retrieval scheme developed by the Library of 
Congress, which has been tested and evaluated for several years. Standardized searching on the Web, adapting the LC 
retrieval model, might help to reduce disorientation, 

Marchionini's (1993) techniques of zooming and browsing also lose validity in Netscape® in how information 
is ultimately presented to users. Zooming and browsing can invoke images of movement in a three-dimensional space. 
As Netscape® is currently designed, a space that may be perceived as three dimensional is flattened to two dimensions. 
The information space is represented on a flat screen without elements of depth. 

This potential “warping” of the learner's mental model can be very disorienting, leading the learner to look for 
“clues” in the environment to re-orient themselves, Alyssa looked to a map to improve her orientation to the 
information available. This coincides with Forsyth's (1988) recommendation of a “multi-sensory” approach for 
facilitating memory — as well as improving user orientation. 

Several options are available, but the theoretical “best” option remains elusive. It is clear, however, that 
guidelines need to be established if we are to optimize use of emerging information systems (Jones, 1993). The 
establishment of guidelines, and maintenance within certain parameters, is crucial if learners are to establish useful 
mental models that assist them in their understanding (Norman, 1983). Until learners are able to engage in natural 
mapping of these information systems (Norman, 1988), and guidelines are established that ensure consistency and 
predictability (Wadlow, 1990), disorientation will persist. 
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Research Question Three: Does perceived self-efficacy affect the strategies employed? 

Perceived self-efficacy affected both the number as well as the types of strategies which were engaged. Those 
who reported mid-to-high perceived self-efficacy (Bill, Marsha: mid; Mick: high) engaged in more strategies and at a 
higher level than Alyssa who reported low perceived self-efficacy. This is consistent with studies examining the 
relationship between self-efficacy and performance (Ashton, 1984; Kinzie & Delcourt, 1991; Murphy et al., 1988), where 
high positive correlations were reported between self-efficacy and performance. 

Perceived self-efficacy not only affects the learner’s interactions, but also perceptions of control. Syst :m 
interactions with Bill and Mick confirm this finding; both had higher levels of perceived self-efficacy and were successful 
in their task. Although Marsha was not successful in her target task, she was able to find useful information; her 
confidence and high perceived self-efficacy helped elicit this result. Alyssa, in contrast to the others, demonstrated lower 
perceived self-efficacy throughout. This, in turn, influenced her interactions and strategies, leaving her unsure of where 
she was or how to proceed, and unable to move beyond basic search strategies. 

Research Question Four Does system knowledge affect the strategies employed? 

System knowledge also affects the strategies used in a hypermedia information system. Participants with low 
system knowledge engaged in more basic search strategies than those with higher system knowledge. 

The level of system knowledge had a stronger influence on strategies used than subject knowledge. This is 
consistent with Park and Hannafin’s (1992) conclusion that the lack of a functional mental model of the multimedia 
system tends to minimize the value of domain knowledge. While Alyssa possessed some domain knowledge, she had no 
prior system knowledge. The absence of a functional mental model, as well as an inability to establish one, resulted in 
very limited success in working with the system and an inability to move beyond basic search strategies. While 
increased system knowledge may not directly lead to success, system understanding is critical to selecting terms and 
knowing how to interact Increased system knowledge can also reduce disorientation and frustration. 

Closely related to a subject being searched is the intent behind what is sought in the system. The way that 
World Wide Web technologies are currently designed makes the environment one of “at your own risk.” Anyone can 
place any kind of information on the Web. In addition, there is minimal structure to the documents. This leaves the user 
in a position where their intent may not match that of the document retrieved. A “best’ solution for this issue remains 
unresolved. 

The ability to use the system influenced the strategies the participants engaged. All participants entered the 
search task with no prior system knowledge directly related to Netscape®. While Mick reported using other hypertext 
systems, none had used Netscape® prior to its introduction in class. The two participants who built a functional model 
for the system. Bill and Mick, were also successful in their tasks and moved beyond basic search strategies. 

Bill and Mick were also the only participants able to describe and depict their mental model of the system. 
Mayer’s (1989) work on conceptual models provides an interesting interpretative background. Mayer found that in studies 
where conceptual models were provided, learners were able to make significant strides in problem solving and 
understanding. While participants in this study were not provided with a model prior to the search task, those who 
generated their own conceptual models engaged in strategies at a problem-solving level and increased their understanding. 

Consistent with previous research, increased system experience and system knowledge seems to affect the 
strategies used in the system (Weil, Rosen, & Wugalter, 1990). The participants reinforced this observation through 
their interactions, and in the stimulated post-search interviews. All stated that, given more time and experience, they 
were confident that they would be better able to use the system. 

Research Question Five: Does subject knowledge affect the strategies employed? 

The level of subject knowledge affected the strategies used in the hypermedia information system. While the 
participants used a variety of strategies independent of their prior subject knowledge, those with lower subject knowledge 
engaged in more primary search strategies. These results assist in confirming what other researchers have found: learners 
with extensive context-related prior knowledge out-perform their cohorts with limited prior knowledge (Langer & 
Nicolish, 1981; Recht & Leslie, 1988). Osman’s (1992) conclusion that the availability of content-related prior 
knowledge as a powerful determinant of retrieved information was also supported in this study. 

The level of prior subject knowledge also affects the ability to integrate and retain new information (Ausubel, 
1963). Bill, who possessed a higher level of subject knowledge, refined his search based on terms he previously tried. 
Marsha, who possessed little prior subject knowledge, refined her search little, using few terms to seek information. 
While the number of terms used during the search process was not a strong indicator of failure, it could be an indication 
of the inability to monitor and judge relevance in relation to information sought. According to Ausubel, the ability to 
integrate and transform information is a crucial component to performance. The lack of integration of the information 
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the user receives while searching for information in Netscape® would cause conflict in relation to performance and search 
success. 



RECOMMENDATIONS AND IMPLICATIONS 

The implications for creating productive and successful OELEs are compelling. Several studies have been 
conducted indicating the promise of these systems (see, for example, Harmon, 1992). However, much work is needed to 
realize the potential of OELEs for learning. 

One area that was a significant influence on the learner’s successful use of the system was a feeling of insecurity 
or discomfort. Hannafin, Hall, Land and Hill (1994) discussed this as a significant issue affecting the use of OELEs. In 
the present study, the second most influential factor inhibiting use of the hypermedia information system was 
disorientation, which led to discomfort and confusion. The methods for overcoming disorientation and discomfort, as 
well as for assisting and supporting learners in their use of the systems, remains unresolved. 

Another issue arising from this study relates to the design and development of OELEs. While computer and 
information scientists have developed design models for creating information systems, and instructional designers and 
psychologists have created design models for the development of learning systems, none seem to satisfactorily support 
the design and development of OELEs. It is an area rich with significant, but unfulfilled, promise. 

The need for guidelines, both in terms of supporting the learner and in developing the systems, is also apparent. 
Guidelines for placing information on the Web and Internet, as well as indexing and providing pointers to this 
information, are not standardized. This lack of standardization makes the development of these systems challenging for 
the designer, and also presents significant challenges to users of the systems as they attempt to create a functional model 
(Norman, 1983) of the information system. While the researcher agrees with Jones (1993) that “hard and fast” rules are 
neither feasible nor desirable for these environments, guidelines are necessary if we hope to see the continued growth and 
utility of these environments. 

A final implication relates to how individuals think. A fundamental problem underlying the lack of 
understanding associated with OELEs is the manner in which “compliant** thinking is shaped by conventional school 
activities (McCaslin & Good, 1992). Traditionally, education has been a lock-step, highly directed experience (Hannafin. 
Hall, Land, & Hill, 1994). This engenders learners who lack the orientation, mental models, and strategies (or 
capabilities for creating them) for these environments (Hill, Lebow, Driscoll, & Rowley, 1994), where divergent 
thinking, multiple perspectives, and independent thinking are critical. 

While educational history was not gathered in the current study, there is evidence to suggest that those learners 
who developed an orientation and functional mental model of the system proved more successful in their task. These 
learners used two primary techniques associated with OELEs: divergent and independent thinking. Adjusting the ways 
we teach to foster divergent thinking and multiple perspectives may assist learners in using these environments 
effectively and with minimal disorientation. 

As open-ended hypermedia information systems continue to grow, both in their capabilities and affordances, the 
need to understand how to support learners in their use of the systems becomes critical. The promise of hypermedia and 
OELEs has been widely heralded (see, for example, Bransford et al., 1986; Grabowski & Small, 1991; Marchionini. 

1988; Hannafin, Hall, Land & Hill, 1994). Continued research in this area will enable the realization this promise. 
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